ALMOST GLOBAL STOCHASTIC STABILITY* 

RAMON VAN HANDELt 

Abstract. We develop a method to prove almost global stability of stochastic differential 
equations in the sense that almost every initial point (with respect to the Lebesgue measure) is 
asymptotically attracted to the origin with unit probability. The method can be viewed as a dual to 
Lyapunov's second method for stochastic differential equations and extends the deterministic result 
in [A. Rantzer, Syst. Contr. Lett., 42 (2001), pp. 161-168]. The result can also be used in certain 
cases to find stabilizing controllers for stochastic nonlinear systems using convex optimization. The 
main technical tool is the theory of stochastic flows of diffeomorphisms. 

Key words, stochastic stability, stochastic flows, nonlinear stochastic control 

AMS subject classifications. 34F05, 60H10, 93C10, 93D15, 93E15 

1. Introduction. Lyapunov's second method or the method of Lyapunov func- 
tions, though developed in the late 19th century, remains one of the most important 
tools in the study of deterministic differential equations. The power of the method 
lies in the fact that an important qualitative property of a differential equation, the 
stability of an equilibrium point, can be proved without solving the equation explic- 
itly. The theory was generalized to stochastic differential equations in the 1960s with 
fundamental contributions by Has'minskh JOj and Kushner |15| . 

Lyapunov's method also underlies many important applications in the area of 
nonlinear control Finding optimal controls for nonlinear systems is generally an 
intractable problem, but often a solution can be found which stabilizes the system. 
Unlike in deterministic control theory, where nonlinear control is now a major field, 
there are very few results on stochastic nonlinear control. It is only recently that 
stochastic versions of the classical stabilization results of Jurdjevic-Quinn, Artstein 
and Sontag were developed by Florchinger [SJ [!]] and backstepping designs for 
stochastic strict-feedback systems were developed by Deng and Krstic [SHE]- 

In this paper we will not consider stochastic stability in the sense of Has'minskh; 
rather, we ask the following question: for a given Ito stochastic differential equation 
on ]R n , can we prove that for almost every (with respect to the Lebesgue measure on 
R") initial state the solution of the equation converges to the origin almost surely as 
t — > oo, i.e., is the origin almost globally stable? This notion of stability is clearly 
weaker than global stability in the sense of Has'minskh, but is of potential interest in 
many cases in which global stability may not be attained. 

Our main result is a Lyapunov-type theorem that can be used to prove almost 
global stability of stochastic differential equations, extending the deterministic result 
of Rantzer The theorem has several remarkable properties. It can be viewed as 
a "dual" to Lyapunov's second method in the following sense: whereas the Lyapunov 
condition reads ££V < 0, where is the characteristic operator of the stochastic 
differential equation and V is the Lyapunov function, the condition that guarantees 
almost global stability reads ££*D < where Jzf* is the formal adjoint of Jz? (also 
known as the Fokker-Planck operator.) Hence the relation between the two theorems 
recalls the duality between densities and expectations which is prevalent throughout 
the theory of stochastic processes. 

•This work was supported by the ARO under Grant DAAD19-03- 1-0073. 
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2 



RAMON VAN HANDEL 



A further interesting property is the following convexity property. Suppose we 
are given an Ito equation of the form 

/t m pt 

(X (x T ) + u(x T 
)Y(x T ))dT + J2 / X k (x T )dW? 
k=l 

where Afc(O) = for k = . . . m and u(x) is a state feedback control. The goal is to 
design u(x) such that the origin is almost globally stable. It is easily verified that the 
set of pairs of functions (D(x),u(x)D(x)) which satisfy 3f*D < is convex. Note 
that the classical Lyapunov condition _Sf V < is not convex. 

The above convexity property was used in the deterministic case by Prajna, Par- 
rilo and Rantzer [22 to formulate the search for almost globally stabilizing controllers 
as a convex optimization problem, provided that X^, Y, D and u are rational func- 
tions. The method applies equally to the stochastic case and thus provides a tool for 
computer-aided design of stochastic nonlinear controllers. 

It must be emphasized that almost global stability is a global property of the flow 
which places very few restrictions on the local behavior near the origin. In particular, 
local stability is not implied 1 . A very fruitful approach to studying the local dynamical 
behavior of stochastic differential equations (and more general random dynamical 
systems) is developed by Arnold 0- First, the flow associated to the stochastic 
differential equation is linearized; then Oseledec's multiplicative ergodic theorem is 
used to provide a suitable "time-averaged" notion of the eigenvalues of the linearized 
flow. To prove almost global stability we do not linearize the flow, though the proofs 
still rely on the flow of diffcomorphisms generated by the stochastic equation. We 
refer to for an introduction to the dynamical approach to stochastic analysis. 

This paper is organized as follows. In fJ2]we fix the notation that will be used in 
the remainder of the paper. In fj^l we reproduce the deterministic result of Rantzer 
[221 with a significantly different proof that generalizes to the stochastic case. In 2] 
we review the theory of stochastic flows of diffcomorphisms generated by stochastic 
differential equations. is devoted to the statement and proof of our main result 
for the case of globally Lipschitz continuous coefficients. In §§\ the main result is 
extended to cases in which the global Lipschitz condition does not neccessarily hold. 
A few examples are given in §J| Finally, in fJSJwe discuss the application to control 
synthesis using convex optimization. 

2. Notation. Throughout this article we will consider (stochastic) differential 
equations in R™. The Lebesgue measure on M. n will be denoted by \i. R+ denotes the 
nonnegative real numbers and Z + the nonnegative integers. 

We remind the reader of the following definitions: for < a < 1, a function 
/ : X — > Y from a normed space (X, || • ||) to a normed space (Y, || • \\y) is called 
globally Holder continuous of order a if there exists a positive constant C such that 

\\f(x)-f(y)\\ Y <C\\x-y\\ a Vx,yeX. (2.1) 

/ is locally Holder continuous of order a if it satisfies the condition l|2.1|l on every 
bounded subset of A. / is called globally (locally) Lipschitz continuous if it is globally 
(locally) Holder continuous of order 1. / is called a C k,a function if it is k times 
continuously differentiable and the k-th derivatives are locally Holder continuous of 
order a for some k € Z + and < a < 1. 

lr The term "stability" seems a bit of a misnomer; despite that almost all points converge to 
the origin, a trajectory that starts close to the origin could move very far from the origin before 
converging to it. We have used the term that has been used in the deterministic literature, e.g. 1181 . 
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3. The deterministic case. In this section we give a new proof of Rantzer's 
theorem 23. which is a deterministic counterpart of our main result. Our proof 
demonstrates the main features of the proof of the stochastic result in the simpler 
deterministic case. 

The following Lemma is similar to Lemma A.l in 23 , and we omit the proof. 

Lemma 3.1. Let f G C 1 (R n ,M") be globally Lipschitz continuous, S C R" be 
an invariant set of x(t) = f(x(t)) and Z C S be [i-measurable. Let D G C 1 (S, R) be 
integrable on Z . Then 



j 

Jth 



D{x)dx = / D{x)dx- / / [V •(/£>)] (x)dxdr (3.1) 
t 1 (z) Jz Jo J4>- X {Z) 



where <p t ■ R" -> R" is the flow of f. 

Theorem 3.2. Let f G C^R^R") be globally Lipschitz continuous and let 
/(0) = 0. Suppose there exists D G C 1 (R"\{0},R+) such that D is integrable on 
{x G R™ : |x| > 1} and [V • (/!?)] (x) > for /i-almost all x. Then for ^i-almost all 
initial states x(0) the solution of x{t) = f(x(t)) tends to the origin as t — > oo. 

Proof Let S = R"\{0}, e > and Z = {x G R n : |x| > e}. Note that 4> t {x) is a 
diffeomorphism for every i £ R; hence </>t(x) is one-to-one, and as </>t(0) = 0, t G R is 
a solution of x(t) — f{x{t)) there can be no x G S such that </>t(x) = for some t G R. 
We have thus verified the invariance of S under the flow <pt{x). We now invoke Lemma 
13.11 As D{x) is nonnegative expression (j3.1|l is also nonnegative. Furthermore, 1)3.1(1 
is finite because D is integrable on Z, and is nonincrcasing due to [V • (/£>)] (x) > 0. 
By monotone convergence the limit as t — > oo exists and is finite. Hence 

r l {Z))dr < oo, B(A)= [ [V • (fD)](x) dx. 



A 

Note that the assumption [V • (/£>)] (x) > implies that D is a measure on S. The 
measure space (S, D) is er-fmite as D({x G S : \ < |x| < k}) < oo for all k > 1 and 
IC= 2 {* e S : ± < |x| < fe} = S. 

We now fix some m G N and divide the halfline into bins S 1 ™ = [{k— 1)2 m , /c2 TO ] , 
fc G N. From each bin we choose a time t™ G S"™ such that 



-k 



-2(Z))< t M n D(^ 1 (Z)) + 2 
For fixed m, we denote this discrete grid by T m = {t™ : A: G N}. We now have 

00 poo 

2- m rB(^(Z)) < 2- m + / P(0- 1 (Z))dr<oo. 
k=i " 7o 

As D is cr-finite we can now apply the Borel-Cantelli lemma, which gives 
D (limsup^(Z) ) = /x (limsup^(Z) J = 

\ k — >oo / \ k — >oo / 

where the first equality follows as [V- (/£>)] (x) > /i-a.e. implies /i <C D. Consequently 
/i ( I J limsup^-^Z) ) <V/i flimsup^^Z)) = 0. 
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We have thus shown that the set of initial states x for which there are, for some m, 
infinitely many times t G T m such that <j>t(%) G Z, has Lebesgue measure zero. 

We now claim that if limsup^^ |<^ t (x)| > e, then we can choose m so that there 
are infinitely many times t in T m such that <f>t{x) G Z . The statement is trivial if also 
liminff^oo \<fi t (x)\ > e; let us thus assume that liminf^oo \<fi t (x)\ < e. We will need 
the following result. Due to the global Lipschitz condition and /(0) = 0, we have 

\M*)\ < \M*)\+ [ \f{Mx))\da< |0.(x)|+C / \M*)\da (3.2) 

J s J s 

for some constant C > 0. Thus Gronwall's lemma gives |<^t(a;)| < \4> s (x)\ e c ^^ s \ 

Now note that liminL^oo \tfi t (x)\ < e < limsup^^ \4>t(x)\ implies that there exist 

e" > e' > e such that (i) there are infinitely many upcrossings of the curve |0t(a;)| 

through e', and (ii) |<At(x)| crosses e" infinitely often. Denote by t" a time such that 

\4>t"(x)\ — e" and by t! the latest time previous to t" that |^t/(ac)| = e' . Then clearly 

t" — t' > ^ log p-. As this happens infinitely often, we conclude that <j>t(x) infinitely 

i // 

often spends a time in excess of ^ log p- in Z. But then clearly m can be chosen large 
enough so that every such interval includes at least one of the t™ G T m . 

We have now shown that for //-almost all x G R n we have limsup^^ |</>t(a;)| < e, 
i.e. for /i-almost all x G R" 3t e > such that |0t(x)| < e for t >t e . But as this holds 
for any e > the trajectories must converge to the origin. □ 

4. Stochastic flows. The purpose of this section is to review, without proofs, 
some results of the theory of stochastic flows of diffcomorphisms generated by stochas- 
tic differential equations. A detailed exposition on the subject can be found in |13U14j 
and shorter treatments are in 0121 El 1X3 ■ 

Throughout this article (fi,-? 7 , P) denotes the canonical Wiener space of the Tri- 
dimensional Brownian motion W* with two-sided time M. We also introduce the 
two-parameter filtration T\ = cr{W^ — : s < v < u < t, 1 < k < m}. The 
extension to two-sided time is important in that it allows us to treat the Wiener 
process as a dynamical system ^ |2 [3] : 

Theorem 4.1. There exists a one-parameter group {9 t : t G R} of measure- 
preserving transformations o/(f2,.F, P) such that Wt{9 s u>) = Wt+ s (to) —W s (u>) for all 
lu G f2 and s,t€l. 

We will consider Ito stochastic differential equations of the form 

X {x T )dr + Y. / X k {x T )dW>; (4.1) 

with the following assumptions: 

1. x G R™. 

2. Xfc : M™ — > M™, fc — . . . m are globally Lipschitz continuous. 

The global Lipschitz condition guarantees many nice properties of the solutions; we 
will assume it for the time being, and later relax this requirement somewhat in ^ 

Denote by £ S) t(x, w) (or simply £ s ,t(x)) the solution of (|4.1J) at time t > s given 
the initial condition x s = x. It is well known that in the case of globally Lipschitz 
continuous coefficients there exists a unique, non-exploding solution £ Sjt (a;) which is 
an .F'-semimartingale and is in L p for any p > 1 (e.g. |13j). 

Theorem 4.2 f |13l l2*]L Suppose X^, k = 0. . . m are globally Lipschitz continuous 
and let s < t. Then we have the following properties: 
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!• £s,s( x >w) = x for all s and u>. 

2. For any u we have f s ,t(-j0uw) = £«+«,*+«(•, k>). 

3. For almost all uj we have £s,t("> — £r,i(£s,r(") ^)) /or all s <r <t. 
4- £ s ,t(x) isf-a.s. continuous in (s,t,x). 

5. For a.e. oj the map £ Sj t(-,o>) : K™ — * W 1 is a homeomorphism for all s < t. 

The following result establishes that, under additional smoothness conditions, 
£ s ,t(x) is in fact a stochastic flow of diffeomorphisms. 

Theorem 4.3 f|13|). Suppose X^, k = . . .m are globally Lipschitz continuous 
and that they are C p,a functions for some p > 1 and < a < 1. T/ien /or almost all 
ui the map £ Sj t(-,u;) : K™ — > K™ is a C p diffeomorphism for any s <t, and 



0=1 Js 



i — i a i ° S 



k=l 0=1 • 

It will be convenient for our purposes to work with the inverse flow £~ t (x), con- 
sidered as a backward stochastic process in the time variable s (with t fixed). This 
will not give rise to ordinary Ito integrals as s behaves like a time-reversed variable, 
and hence the adaptedness of the process runs backwards in time. The Ito backward 
integral is defined as pi] 

/t _ n— l 

fM ee lim in prob £ ft k+1 (W tk+1 - W t „) 
k=o 

where f s is a backward predictable process with J"' \ f u \ 2 du < oo a.s., and the formal 
construction of the integral from simple functions proceeds along the usual lines. 
The backward integral has similar properties to the forward integral; in particular, 
it is a backward JF^-local martingale (for fixed t) and satisfies an Ito formula (e.g. 
U, pp. 124) which is proved in the same way as its forward counterpart: given 
£s = £t + / a a da + J2k f s (Pa)k dWj? with backward predictable processes a s , (b s )k 
s.t. f a a do < oo a.s., f \(b a )k\ 2 do < oo a.s., then for any C 2 function F : M™ — » K 



pas) = f(6) + 2 E E / MMa^cw d(j 

: ap ra n „t 0171 



fe=l i,j=l ' 



We can now formulate the following result. 

Theorem 4.4 f|13p. Suppose Xk, k = 0...m are globally Lipschitz continuous 
and that they are C p ' a functions for some p > 2 and < a < 1. Then 



C}(x) =x- Xofcfa)) do - / Xkfcfa)) dW a 

J S i i J s 

where we have defined 



fe=i ' 



m 71 t~y 

X (x) = X (x) -EE X k( x )Q^ X k( x )- 
k=l 0=1 



G 
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This expression can be manipulated much in the same way as its forward coun- 
terpart. In particular, under the conditions of Thcorcm l4 . 41 and using i|4.3|) . we obtain 
for any C 2 function F : R™ — ► R the backward Ito formula 

k,i,j Js X X 

/t ac 1 ft , 

^o(ciW)^-(ciW)^-E/ (4.4) 

Similarly we can differentiate the inverse flow, giving 

den*) 1 ri a r 0x^,-1, ^dchxf 



E/ to?^.*^))-^-^ 

fe=l / 3=l I 



This expression is obtained, in the same way as its forward counterpart 14. 2L by 
letting y — > in the backward expression corresponding to |13| . pp. 219, Eq. (4). 

5. The main result. We consider an Ito equation of the form (|4.1|) . We write 



m n 



**f( x ) - E ^j(^w^w/W)-E^j( x oW/W)- 

fc— 1 ij — l i—l 

The following is our main result: 

Theorem 5.1. Let Xf., k — . . .m be globally Lipschitz continuous and C 2 ' a for 
some a > 0, and let Xf.(0) — 0. Suppose there exists D G C 2 (R™\{0}, R+) such that 
D is integrable on {x G R™ : \x\ > 1} and J£*D{x) < for fi-almost all x. Then 
for every initial time s and fi-almost every initial state x the flow £ s ,t(%) tends to the 
origin as t — ► oo P-a.s. 

Before we prove the theorem, let us prove a stochastic version of Lemma \'S. II 
Lemma 5.2. Suppose Xk, k = 0...m are globally Lipschitz continuous and 
C 2,a functions for some a > 0. Let Si C SV+i C R™ be an increasing sequence of 
open sets such that T£ — sup{s < t : £~l(x) G" Sg} — > — oo as I — > oo P-a.s. for 
every x G S — {JpS(. Suppose there is a D G C 2 (S, R+) i/iat is integrable on a 
measurable set Z G S , that obeys J??*D < on S, and such that for each £ there is a 
Di G C 2 (R",R+) that coincides with D on Si. Then 

0< / D(x)dx+ f E / £?*D(x)dxdo- 
Jz Js J(-^(Z) 

for all s < t, and in particular the limit as s — > — oo o/ ife's expression is well defined. 
Proof. Denote by 3 St t{x) the matrix with elements J Si t(x)j = d£~l (x) % / dap , i.e. 

^(UW)J^H a ^-E / ^»))^(*)?«^ 
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by Eq. 14. 5|) . Denote by |J S t (a;)| its determinant, i.e. 



E • 

J'l— Jn = l 



where e, 



is the antisymmetric tensor. Using Ito's rule and straightforward cal- 



culations we obtain 



dX, 



3 s , t (x)\ = l-J2 I -Q^& x ))\ 3 M\dv-J2 I ^|(ClW)|J, t ^)K 



4 dXl 



dx 1 



ax 

dx 



dx 1 



\3 a< t[x)\ da. 



Note that as £7t (') is a diffeomorphism a.s., its Jacobian J s ,t(-) must a.s. be an 
invertible matrix; but as |J Si t(a:)| has a.s. continuous sample paths and |Jt t t(a;)| = 1, 
this implies that a.s. |J Sj t(x)| > for all s < t. Using (|4.4|) with F — Dg and Ito's 
rule we obtain 

< D t fci(x))\J., t (x)\ = D t {x) + j\^*Dg){^}{x))\3^{x)\da 

m n r f 



EE 

k=l i=l 



dx 1 



(Cl(x))\J„Ax)\dW a k . 



Now note that as Dg coincides with D on Si, we can identify (Jz? * Dg){£ iS y Te t {x)) = 
{££* D)(£J V t( x )) f° r every £. Moreover, as the last term in the expression above is a 
backward local martingale, there exists a sequence of stopping times r' p \ — oo such 
that the stochastic integral stopped at r p is a martingale. Replacing s by s V Tg V r' p 
in the expression above and taking the expectation gives 



< D(x) +) 



^)(C t 1 ( 2 ;))|J CT , t (x)|d ( 7. 



We can now let £,p — > oo by monotone convergence. Integrating both sides gives 



< / D(x) dx + 



(J?*D)(Z-l(x))\J a , t (x)\dxda 



where we have used Tonelli's theorem to change the order of integration. The result 
follows after a change of coordinates. □ 

The proof of Theorem 15. II is similar to the proof of Theorem 13. 21 The stochastic 
version of the argument following l|3.2(l . however, is a little more subtle, as we do not 
have a pathwise upper bound on the rate of growth of sample paths. On the other 
hand, we can establish such a bound in probability which, together with the strong 
Markov property, is sufficient for our purposes; a similar argument was used in 
to the same effect. For this purpose we give the following Lemma, various versions of 
which appear in the literature (the result below is adapted from 

Lemma 5.3. Let X^, k — . . . m be locally Lipschitz continuous and A > 0. Then 



sup \£,s. s+ s(x) - x\ > A 

0<5<A 
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where Ki,K% < oo are constants that depend only on X and \x\. 

Proof. Let W t be the m-vector with elements W t and X(-) be the n x m-matrix 
with entries Xl(-), k = 1 . . .m. For r > 0, define B r (x') = {x E W 1 : \x — x'\ < r}. 
B r = B r (0), and 



po(r) = sup \X (y)\, P i(r) = sup ||X(y)|| = sup tr[X(j/) T X 

\y\ < r \y\ < r \y\ < r 



■(»)] 



1/2 



Let r r be the first exit time of £ s ,t(z) from £? r . In 0, pp. 1240 it was established that 



E 



SUP \£, s ,(s+8)/\t,,( x ) ~ x \ 2 
0<S<A 



Hence we have by Markov's inequality 



sup \is,( s +s)AT r ( x ) ~ x \ t A 

0<S<A 



<2p (r) 2 A 2 + 8 Pl (r) 2 A. 



< A- 2 (2p (r) 2 A 2 + 8 Pl (r) 2 A). 



Now note that B\(x) is strictly included in B\ x \ + 2\i so that the first exit time from 
B\(x) is no later than t\ x \ + 2\- But then the events 



sup |£ Si(s+(5)Ar {x)-x\>X\ 

0<<5<A J 



w : sup |^ SiS+ i(a;) - x| > A 

0<5<A 



are equivalent; after all, the events are equivalent on TU1+2A > s + A by construction, 



whereas if ti 



\x\+2X 



< s + A both events must be true as |£ s 



x\ > X. Hence 



sup |£ SjS+(5 (x) - x\ > X 

0<S<A 



< A- 2 (2p (|a;| + 2A) 2 A 2 + 8pi(|x| + 2A) 2 A) 







where we have set r = \x\ + 2A. This completes the proof. 
We now turn to the proof of the main theorem. 

Proof of Theorem l5~fl Let e > and Z = {x £ R" : |x| > e}. We begin by 
applying Lemma IB~2l To this end, define Si = {x 6 M™ : \x\ > i^ 1 }, so S — {J e Si = 
]R™\{0}. Clearly D is integrable on Z and there exists a C 2 (R",R+)-approximation 
Di of D for each I. It remains to check that ti — > — oo. Suppose that this is not the 
case; then given x S S there must be a positive probability that £7t( x ) = f° r some 
— oo < s < t. But £7* (0) = f° r au s an d a - s - Cs"t i x ) is one-to-one for all s < t, so 
this cannot happen. Hence all the conditions of Lemma l5~2*l are satisfied, and we have 



0< / D{x)dx+ [ E f ^f*D{x)dxda. (5.1) 

Now note that (|5.1|l is nonincreasing with decreasing s due to Jz? *D < and is finite 
because D is integrable on Z . By monotone convergence the limit as s — > — oo exists 
and is finite. Hence 

f B(^ t {Z))da < oo, B(A) = - f £f*D(x) (P(du>) x ^(dsc)) 

where we have used Tonelli's theorem to convert the iterated integral to a single 
integral with respect to the product measure, and we slightly abuse our notation by 
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writing = B({(u,x) £ ilxS : ^ t {x,ui) G Z}). Note that ££*D < implies 

that D is a measure on x S, and ED is cr-finite as D(0 X {x € S ; 1- < \x\ < k}) < oo 
for all k > 1 and U r ? =2 ( fi x e 5 : \ < \x\ < k}) = x 5. 

We now fix some m G N and divide the halfline into bins 5™ = [(k— l)2 _m , fe2 -m ], 
fc G N. From each bin we choose a time i™ € 5™ such that 

D(cV,t(^)) < in i ^(^4*^)) + 2 ~ fc - 

For fixed m, we denote this discrete grid by T m — {t™ : k G N}. We now have 

2-™ jr D(C_V >t (£)) < 2- 1 + /" d<7 < oo. 

k=i J -°° 

Using the fact that the transformation 6 t of Theorem 14. II is P-preserving to shift the 
times £J™ to the forward variable, we obtain 



fe=i k=i 

As B is cr-finite we can now apply the Borel-Cantclli lemma, which gives 
Iimsup ) = (P x ,i) ( limsupf-i (Z) ) = 

where the first equality follows as ££*D{x) < fi-a..e. implies Px(i<cB. Consequently 

(P x M ) ( Q limsu P C s 1 +t (^) j < £(P x M ) (hmsup^+tC^)) = 0- 
\m=l teTm / m=l ^ * 6Tm ' 

We have thus shown that for all initial states x, except in a set N C R" of Lebesgue 
measure zero, there is P-a.s. for any m only a finite number of times t in the discrete 
grid T m such that £ SyS+t (x) G Z. 

Let us fix an x $ N. We now claim that the fact that P-a.s. for any m there 
is only a finite number of times t G T m such that £ S)S _|_t(a:) £ Z implies that P-a.s. 
limsupj^^ |£ s ,t(x)| < e. To see this, suppose Ppimsup^^ \£, s ,t{x)\ > e] = 5 > 0. 
By monotone convergence E[xiimsup |£ s t (x)|>e'] <5 as e' \ e, hence there exists an 
e' > e such that Ppimsup^^ |£ Sj t(a:) > e'} > 0. We have already shown, however, 
that a.s. |£ s ,t(x)| < £ for infinitely many times t n /* oo. Hence 



limsup |£*,t(aO| > e' 

£ — >oo 



> ==> P[|£ Sii (a;)| crosses e and e' infinitely often] > 0. 



Once we disprove latter statement, the claim is proved by contradiction. 

To this end, introduce the following sequence of predictable stopping times. Let 
<jq = mi{t > s : |£ Sj t(a;) < e}, tq = mi{t > <jq : \£, s ,t[%)\ > £'}, and for any n > we 
set <7„ = inf{t > r„_i : |£ s ,t(x)| < £ }> T « = mf {^ > : |f«,t(aOI > e'}. Define 

Q n (A) = {u e O : r n < oo, |6,t„+<s(x)| > e V0 < 5 < A}. 

For any A > 0, the set of cj G 51 such that a; G f2 n (A) for infinitely many n must 
be of P-measure zero; after all, we can choose m sufficiently large so that every time 
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interval of length A contains at least one point in T m , and for points t £ T m we have 
|£«,t( x )l > £ only finitely often P-a.s. Thus E n Xn„(A) < oo P-a.s. To proceed, we 
use the following argument (see pp. 398-399). Introduce the discrete filtration 
B k = T T s h+1 and define Z k = X k - Y k with 

x k = S n= iXn„(A)) Y k = E„=i E [xn„(A)|$n-i]- 

As ftfc(A) £ B n for all k < n, Z k is a Z^-martingale. Now define for a > the 
stopping time n(a) = infjrt : Z n > a}. As \Z k — Z k -\\ < 1 a.s., the stopped process 
Z' k = Zf. AK r a \ is a martingale that is bounded from above, and by the martingale 
convergence theorem Z' k converges a.s. as k — > oo to a finite random variable Z'^. 
But as Z' k and Z k coincide on {lo : sup n Z n < a} and a > was chosen arbitrarily, 
we conclude that Zj. — > < oo on {u : sup n Z„ < oo} (modulo a null set). Note, 
however, that X n and Y n are both positive increasing processes and we have already 
established that sup n A„ < oo P-a.s., so sup n Z n < oo P-a.s. But this implies that 
Zk, and hence also Y k , converges to a finite value P-a.s. Thus we have established 

En=i E [Xn„(A)i^T"] <oo P-a.s. for any A > 0. 

Note that by the continuity of the sample paths |£ s ,t„ (•"£)! = s' on r„ < oo. By Lemma 
15.31 we can choose A > sufficiently small such that 



P(y) 



£' — 

sup_|£ SjS+<5 (y) -y\< — — 

0<<5<A Z 



1 

> - 

- 2 



for all \y\ = e' . Using the strong Markov property, we can write 

00 > E^l E [Xf2„(A)l- 7 7 n ] ^ T,n=l P (&,r n ( X )) Xr n <oo > ^T,n=lXr n <oo V-&.S. 

But this implies that r„ < oo finitely often P-a.s., contradicting the assertion that 
Pll&j.tCz)! crosses e and e' infinitely often] > 0. This is the desired result. 

We have now shown that for /^-almost all x £ R", P-a.s. limsup^^ |£ s ,t(a;)l < £, 
i.e. for /i-almost all x £ R™ P-a.s., 3t e > s such that |£ s ,t(aO| < e for t > t e . But as 
this holds for any e > the flow must converge to the origin. □ 

The proof of Theorem 15. II is readily extended to prove other assertions, such as 
the following instability theorem. 

Theorem 5.4. Let X k , k = 0...m be globally Lipschitz continuous and C 2,a 
for some a > 0. Suppose there exists a D £ C 2 (R™,K+) such that Jzf*D(x) < for 
^-almost all x. Then for every initial time s and [i-almost every initial state x the 
flow escapes to infinity, i.e. \£ s ,t(%)\ — > oo as t — > oo P-a.s. 

Proof. Let e > and Z' = {x £ M™ : |x| < e}. Again we begin by applying 
Lemma T5.2I We can simply choose Sg = S = M. n for all I; by non-explosion 7> = — oo 
and the remaining conditions are evident. Hence 

0< / D(x)dx+ [ E f 5?*D(x)dxda. 

JZ' Js ^Ct( z ') 

Proceeding in exactly the same way as in the proof of Theorem 15. II we can now show 
that for /i-almost all x £ R", P-a.s. liminf^oo |£ s t(aO| — £ > i- e - f° r /^-almost all x £ W 1 
P-a.s., 3t e > s such that |£ s ,t(aO > £ for t > t e . But as this holds for any e > the 
flow must escape to infinity. □ 
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Remark. At first sight the statements of Theorems 15.11 and 15.41 may seem con- 
tradictory, but this is not the case. The essential difference between the theorems is 
the region in R" on which D is integrable. Roughly speaking, the idea behind the 
proofs of Theorems 15.11 and 15.41 is to show that if J??*D < /i-a.e., then the solution 
of the Ito equation can only spend a finite amount of time in any region on which D 
is integrable. Hence in Theorem 15 . II the solution will attract to the origin, whereas in 
Theorem 15 .41 the solution attracts to infinity. 

If we try to satisfy the conditions of Theorems 15 . II and 15 .41 simultaneously we will 
run into problems. Suppose we have a nonnegative D G C 2 (R"), as in Theorem 15.41 
which is integrable as in Theorem l5.ll Then D is a normalizable density function, i.e. 
we could normalize D and interpret it as the density of the Ito equation at some point 
in time. But then Jz?*D < would imply that the associated Fokker-Planck equation 
does not preserve normalization of the density. Evidently Theorem 15.41 can only be 
satisfied if D is not integrable, whereas Theorem 15 . II requires D to have a singularity 
at the origin. See |7|for examples. 

6. Further results. In this section we extend the main result to cases in which 
the global Lipschitz condition is not necessarily satisfied. We first show that the result 
of Theorem 15.11 still holds if we can convert the coefficients of (|4.1|) to be globally 
Lipschitz continuous through a suitably chosen time transformation. In particular, 
this allows us to treat the case that X k , k — ... m and their first derivatives are 
polynomially bounded, provided that some additional intcgrability conditions on D 
are satisfied. We also extend the main result to the case in which the flow is restricted 
to an invariant subset of R ra with compact closure. 

Theorem 6.1. Let X k : R" — » R n , k = 0...m be measurable and let X k (0) = 0. 
Suppose there is a strictly positive measurable map c : R™ — > (0, oo) such that c{x) and 
c(x)~ 1 are locally bounded, and such that c(x)Xq(x) and y/ c{x) X k (x), k — l...m 
are globally Lipschitz continuous and C 2 ' a for some a > 0. Suppose there exists 
D : R"\{0} -> R+ such that D(x)/c(x) is C 2 , is integrable on {x 6 R" : |x| > 1}, 
and S£*D{x) < for ^.-almost all x. Then for every initial time s and fi-almost every 
initial state x the solution x t of (|4.1(l tends to the origin as t — > oo P-o.s. 

Proof. Consider the Ito equation 



We will write Y (y) = c(y)X (y), Y k {y)_=_^cjy) X k {y) [k - l...m), and D(y) = 
D(y)/c(y). Note that by construction J£*D(y) = J£*D{y), where _Sf* is the adjoint 
generator of (|6.1f) . By our assumptions we can apply Theorem l5.1l to the Ito equation 
IjB.ip. Thus for all y s G R", except in a set N with (j,(N) = 0, y t -> as t — > oo P-a.s. 
Now choose any y s $ N and define 



Note that a T is an ^-stopping time for each r. We claim that (it — > oo as t — > oo a.s.; 
indeed yt a.s. spends an infinite amount of time in an arbitrarily small neighborhood 
of the origin, and as c(x)~ l is locally bounded c{x) > 6 > in any such neighborhood. 
Moreover, /3 t < oo a.s. for any t as c{x) is locally bounded, and hence at — > oo as 
t — > oo a.s. From §V.26 (pp. 175) it follows that the time rescaled solution y at 




(6.1) 
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is equivalent in law to the solution x t of Q4.ip . But as almost all paths of the process 
y t go to zero asymptotically and as a t — > oo a.s., the result follows. □ 

COROLLARY 6.2. Suppose that all the conditions of Theorem I5.il are satisfied 
except the global Lipschitz condition. Suppose that additionally X k , k = 1 . . . m satisfy 
\X k {x)\ < C k (l + \x\P +1 ), \dX k {x)/dx'\ < C' k (l + \x\P), and \X (x)\ < C (1 + \x\ 2 p +1 ), 
\8Xq (x)/dx l | < Cq(1 + \x\ 2p ) for some p > 1 and positive constants C k ,C' k < oo. If 
(1 + \x\ p ) 2 D(x) is integrable on {x G R" : |x| > l}, then Theorem \5.1\ still holds. 

Proof. Let c(x) = (1 + \x\ p )~ 2 , and note that c{x) is smooth, strictly positive 
and that c(x) and c(x)~ 1 are locally bounded. Let Yq(x) — c(x)Xq(x) and Y k (x) = 
\/c(x) X k (x), k = l...m; as \/c(x) is smooth and X k , k = 0...m are C 2,a , the 
coefficients Y k are also C 2a . To show that Y k are globally Lipschitz continuous 
it suffices to show that their first derivatives are bounded. Let us write y k ^(x) — 
\dY k /dx l \(x). First consider the case k > 1. Then 



Vk,i(x) < 



1 + \x\p 



dx k (x) 



dx l 



(l + \x\P) 2lXk{X > l - Ck + Ck (l + \x\P) 2{1 + lxl > 



which is bounded. Similarly, 



s c « (ttW (i + ixfp) + c °W3f (i + w2p+1) 

is bounded. Finally, D(x)/c(x) = (1 + |a;| p ) 2 L>(a;) is C 2 , as is C 2 and c(x)" 1 is 

smooth on R™\{0}. Hence, provided D(x)/c(x) is integrable on {x G M™ : |x| > 1}, 
we can apply Theorem 16. II □ 

We now turn our attention to stochastic differential equations which evolve on an 
invariant set. The following notion of invariance is sufficient for our purposes: 

Definition 6.3. A set K is called backward invariant with respect to the flow 

£ M ifC,t( K ) c K a - s - f° r al1 s < t - 

We can now formulate the following result. 

Theorem 6.4. Suppose the ltd equation (|4.1|) evolves on a backward invariant 
open set K with compact closure K . Let X k , k = . . . m be C 2 a for some a > and 
let X k (0) = 0(EK. Suppose there exists D e C 2 (K\{0}, R+) such that S£*D{x) < 
for /j,-almost all x £ K . Then for every initial time s and [i-almost every initial state 
x G K the flow £, a j{x) tends to the origin as t — > oo V-a.s. 

Proof. We will assume without loss of generality that X k , k = . . . m are globally 
Lipschitz continuous. Indeed, we can smoothly modify X k outside K to have compact 
support without changing the properties of the flow in K, and as X k are already locally 
Lipschitz continuous their modifications will be globally Lipschitz continuous. 

Let e > and Z = {x G K : \x\ > e}. D is integrable on Z, as D is bounded on 
Z and Z has compact closure. Let Si be an increasing sequence of open sets whose 
closure is strictly contained in S = K\{0}, such that 1J £ Si = S. Then there exists 
a C 2 (R™,R + )-approximation Dg of D for each £, obtained by smoothly modifying 
D outside Sg so that its support is contained in K. That T£ — > — oo follows from 
backward invariance and from the one-to-one property of the flow. Hence all the 
conditions of Lemma 15.21 are satisfied, and we have 

0< [ D{x)dx+ [ E [ <£*n{x)dxdo. 
The remainder of the proof proceeds along the same lines as Theorem 15 .11 □ 
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7. Examples. 

Example 1. Consider the ltd equation 

dx t = (x 2 t - 2x t - z 2 ) dt + x t dW t , 
dz t — 2z t (xt — 1) dt + Zt dWt- 

Note that the line z = is invariant under the flow of l|7.1|l . where the solution (x t , 0) 
for an initial state (xo,0) is given by 

dxt = (x t — 2xt) dt + xt dWt- 

This equation has an explicit solution (see also [2] for a detailed analysis of the dy- 
namical behavior of this system) 



Xf 



x Q e- 2t e w *-^ 



1 — xq f Q e 2s e Ws 2 s ds 
Clearly Xt(u>), lu £ f2 explodes in finite time if 

x Q > e- 2s e w °-^ s d^j < oo. 

Hence the system (17.1(1 is certainly not globally stable. 

Nonetheless, almost all points (xq,zq) € M 2 are attracted to the origin. To show 
this, apply Corollary 16.21 with 

1 3 

jD (- T ' Z ) = ( 2 i 2^2 ' Sf*D(x, z) = — < 0. 

[x + z i y [x + z i y 

Hence for a.e. (x Q , z ) e ]R 2 , a.s. (x t ,z t ) — > (0, 0) as t — > oo. 
Example 2. Consider the Ito equation 

dx t — 12 (2z t — l)x t z t dt — |a;t + (1 — 2z t )x t dW t , 

dy t = -\y t dt + {l-2zt)y t dW t , (7.2) 

dz t = -12z t a:j + 2(1 - z t )z t dW t . 

Let Rt = 2z t — 2z\ ~ x\ — y\. By Ito's rule we have 

dR t = -4(1- z t )z t R t dt + 2(1- 2z t )R t dW t . 

Evidently the ellipse {(x, y, z) G M 3 : 2z — 2z 2 — x 2 — y 2 = 0} is invariant under (|7.2|l . 
Local uniqueness of the solution implies that the interior of the ellipse is also invariant. 
Hence K = {(x, y, z) 6 M 3 : 2z — 2z 2 — x 2 — y 2 > 0} is a (backward) invariant set for 
the system l|7.2f) . Consider 

1 12x 2 
D(x,y,z) = —, Jtf*D(x,y,z) = — < //-a.e. 

Hence by Theorem lfi.4l for a.e. (xo,yo,zo) € T^T, a.s. (xt,yt,Zt) — » (0,0,0) as t — » oo. 
Note that (0,0,0) is certainly not globally stable: it is easily verified that any point 
with xq = and zq ^ is not attracted to (0, 0, 0) a.s., as in this case Zt has a constant 
nonzero mean. 
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Example 3. We consider again the system (|7.2[) , but now we are interested in the 
behavior of points in the invariant set K' = {(x, y, z) £ R 3 : 2z — 2z 2 — x 2 = 0, y = 0}. 
As K' is not an open set we cannot apply Theorem 16.41 to study this case. 

Define the transformation (x, z) i— > p = x/z. Note that p is the stereographic 
projection of (x,y,z) G K 1 which maps (0,0,0) i— > oo. As the fixed point (0,0,0) 
cannot be reached in finite time, we expect that the stereographic projection gives a 
well-defined dynamical system on R. Using Ito's rule and 2z — 2z 2 — x 2 — we obtain 
the Ito equation 

/3 20 \ 
dpt = ( g + 2 +p2 ) P* * ~ Pt dWt - 

Note that this expression satisfies a global Lipschitz condition. Now consider 

42 



D{p) = v^T^, STD(p) = - (2 + p2)3/2 < 0. 

Hence by Theorem 15 .41 for a.e. po £ K, a.s. pt — > oo as t — > cxd. This implies that the 
point (x, y, z) = (0, 0, 0) is almost globally stable in K' . 

8. Application to control synthesis. Consider an Ito equation of the form 

(X (x T ) + u T X Q (x T )) dr + Y, X k {x T ) dW? (8.1) 

k=i Js 

where Xq and Xk, k = . . .m are C 2,a for some a > 0, -Xfc(0) = and u t is a scalar 
control input. We consider instantaneous state feedback of the form u t — u(x t ) where 
u(x) is C 2 '° for some (3 > and u(0) = 0. Then by Theorems 15 . 1 1 IET41 or by Corollary 
16.21 x t — > as i — > oo a.s. for a.e. Xq if there exists a D(x), with additional properties 
required by the appropriate theorem, such that 

m n q2 

fc=ii,i=i (82) 

- E ^I^oW^W + Xj(a;)u(a!)£»(a;)) < /.-a.e. 

i— 1 

Note that this expression is affine in D{x) and u(x)D{x) and the set of functions 
(D(x),u(x)D(x)) which satisfy Ij8.2|l is convex. This fact has been used in the deter- 
ministic case by Prajna, Parrilo and Rantzer [221 to search for "almost stabilizing" 
controllers for systems with polynomial vector fields using convex optimization. As 
the stochastic case enjoys the same convexity properties as the deterministic Theo- 
rem this approach can also be applied to find stabilizing controllers for stochastic 
nonlinear systems. Note that that convex optimization cannot be used to search for 
globally stabilizing controllers using LaSalle's theorem as LaSalle's convergence 
criterion j!6l Ija] is not convex. 

The purpose of this section is to briefly outline the method of [221 f° r the synthesis 
of stabilizing controllers using convex optimization. We will also discuss a simple 
example. 

Suppose that Xq and Xk , k = . . . m are polynomial functions (the case of ratio- 
nal functions can be treated in a similar way.) Consider D[x) and u(x) parametrized 
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in the following way: 

t^/ \ a ( x ) , \ c(x) ,„ „. 

Here b(x) is a nonnegative polynomial which vanishes only at the origin, a(x) is a 
polynomial that is nonnegative in a neighborhood of the origin and is such that u(x) 
is C 2 '" , c(x) is a polynomial that vanishes at the origin and 7 > is a constant. The 
orders of the polynomials and 7 can be chosen in such a way that D(x) satisfies the 
integrability and growth requirements of Corollary 16. 21 For fixed b{x) and 7 consider 
the expression 

-b{x) 1+2 ^*D{x) > /Lt-a.e. (8.4) 

with D{x) and u{x) given by l|8.3|) and _Sf * given by l|8.2|l . Then (|8.4|l is a polynomial 
inequality that is linear in the polynomial coefficients of a(x) and c{x). Our goal is 
to formulate the search for these coefficients as a convex optimization problem. 

Verifying whether (|8.4|l is satisfied comes down to testing nonnegativity of a 
polynomial (a nonnegative polynomial can only vanish on a finite set of points, and 
hence is positive /i-a.e.) This problem is known to be NP-hard in general; however, 
a powerful convex relaxation was suggested by Parrilo [201 • Instead of testing H8.4(l 
directly we may ask whether the polynomial can be written as a sum of squares, i.e. 
whether — 6(:c) 7+2 Jz? *D(x) — '}^ i Pi{x) 2 for a set of polynomials Pi(x). The power 
of this relaxation comes from the fact that every sum of squares polynomial up to a 
specified order can be represented by a positive semidefinite matrix; hence the search 
for a sum of squares representation can be performed using semidefinite programming. 
As (I8.4|) is convex in a(x) and c(x) the following is a convex optimization problem: 

Find polynomials a(x), c(x) such that — 6(a;) 7+2 ^f *D(x) is sum of squares. 

This type of problem, known as a sum of squares program, can be solved in a highly 
efficient manner using the software SOSTOOLS [2Ij. We refer to [2DI EH 122] for 
further details on the computational technique. 

Remark. Note that a(0) and c(0) depend only on the value of the constant 
coefficient of the polynomials a(x) and c(x). Thus c(0) = can easily be enforced 
by fixing the constant coefficient of c{x). To make sure a{x) is nonnegative near the 
origin and u{x) does not blow up we can, for example, require a(x) to be of the form 
A + p(x) with A > and p{x) is a sum of squares that vanishes at the origin. 

Note that if the Ito equation (|8.1() evolves on an invariant open set K with com- 
pact closure then the sum of squares relaxation is overly restrictive. A related relax- 
ation that only guarantees polynomial nonnegativity on K for the case that if is a 
semialgebraic set is considered in e.g. [T$| . 

Example. The following example is similar to an example in |22| . Consider the 
Ito equation 

alx t = {2x1 + asflfa " Qx t y 2 + 5y 3 t ) dt + (x 2 + y 2 ) dW t 
dy t = u t dt- (x 2 +y 2 )dW t . 

We choose b(x,y) = x 2 + y 2 and 7 = 2.5. Using SOSTOOLS we find a solution 
with controller of order 3 and a constant a(x). Note that these choices satisfy the 
integrability requirements of Corollary 16. 21 We obtain a stabilizing controller 

u(x) = -2.7 x 3 + 4.6x 2 y - 6.7 xy 2 - 3.4 y 3 
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where 

-{x 2 + y 2 ) 4 - 5 ^*(x 2 + y 2 )- 2 - 5 = 0.35 y 6 - 0.0015 xy 5 

+ 0.6 x 2 y 4 + 0.0026 x s y 3 + 0.33 x 4 y 2 + 0.004 x 5 y + 0.13 x e 

is a sum of squares polynomial. 

Remark. A drawback of this method is that b(x) and 7 must be fixed at the 
outset. We have found that the method is very sensitive to the choice of b(x) and 7 
even in the deterministic case; often an unfortunate choice will cause the search to 
be infeasible. Moreover it is not clear, even if there exists for polynomial Xo, Xk a 
rational u which almost globally stabilizes the system, that a rational D can always 
be found that satisfies ££*D < 0. Nonetheless the method can be successful in cases 
where other methods fail, and as such could be a useful addition to the stochastic 
nonlinear control engineer's toolbox. 
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